Decoding Virtual Agent's Emotion and Strategy from Brain Patterns

Recent advances in technology have paved the way for humanagent interactions to become ubiquitous in our daily lives, and decades worth of research on virtual agents have enhanced these interactions. However, for the most part, the effect of different types of agents on the human brain is unknown, and the neuroscience of human-agent interactions is rarely studied. In this study, we examine the underlying neural systems involved in processing and responding to different types of negotiating agents. More specifically, we show that different brain patterns are observed for various types of virtual agents; consequently, we can decode the strategy and emotional display of the agent based on the counterpart’s brain activity. Using fMRI data, we analyzed participants’ brain activity during negotiations with agents who show three different emotional expressions and use two different types of negotiation strategies. We demonstrate that, using Multi-Voxel Pattern Analysis, we can reliably decode agents’ emotional expressions based on the activity in the left dorsal anterior insula, and also agents’ strategies based on the activity in the frontal pole.